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Abstract -We study the storage of multiple phase-coded patterns as stable dynamical attractors 
in recurrent neural networks with sparse connectivity. To determine the synaptic strength of 
existent connections and store the phase-coded patterns, we introduce a learning rule inspired 
to the spike-timing dependent plasticity (STDP). We find that, after learning, the spontaneous 
dynamics of the network replay one of the stored dynamical patterns, depending on the network 
initialization. We study the network capacity as a function of topology, and find that a small- 
world-like topology may be optimal, as a compromise between the high wiring cost of long range 
connections and the capacity increase. 
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The capacity to memorize and recall multiple items of 
information is fundamental to normal cognition. Recent 
researches suggest that brain operates as a complex non- 
linear dynamic system, and synchronous and phase-locked 
oscillations may play a crucial role in information process- 
ing, perception, memory, and sensory computation [T}{5]. 
There is increasing experimental evidence that informa- 
tion encoding may depend on the temporal dynamics be- 
tween neurons, namely, the specific phase alignment of 
spikes relative to rhythmic activity across the neuronal 
population (as reflected in the local field potential) [5HT2] . 
Indeed phase-coding, that exploits the precise temporal 
relations between the spikes of neurons, may be an effec- 
tive general strategy to encode information in the cortex 

Q3H2DI. 



The importance of precise timing of the neuron activity 
is also suggested by the rule that controls the potentia- 
tion or depression of synaptic strengths, namely the spike- 
timing dependent plasticity (STDP) [2T1I22] . based on the 
precise order and time interval between the pre- and post- 
synaptic spikes, on a window of tens of milliseconds. This 
kind of plasticity, with acute sensitivity to temporal or- 
der, has been demonstrated in various neural circuits over 
a wide spectrum of species, from insects to xenopus laevis 



frog, rodents and humans 

The STDP is strongly asymmetric in the order of arrival 
of pre- and post-synaptic spikes, usually determining a po- 
tentiation in the case of causal order (pre-before-post), and 
depression in the reverse order. This temporal asymmetry 
results in asymmetric connections between neurons, that 
is a crucial ingredient to give rise to dynamical patterns, 
as opposed to static patterns characteristic of symmetric 
connections as in the Hopfield model. 

Another crucial feature of the network is its connectiv- 
ity and topology. Whereas all-to-all, or random, wiring 
is usually assumed in many models, this is possible in 
a tissue culture involving dozens of neurons, but it be- 
comes less and less feasible when millions of neurons are 
involved, owing to space and energy supply limitations. 
The topology of the brain connectivity has been designed 
by evolutionary goals as a compromise between the abil- 
ity to achieve complex dynamical functions and physical 
constraints and costs minimizations [251l26j . 

In the last decade, there has been a growing interest 
in the study of the topological structure of the brain net- 
work. There is increasing evidence that the connections 
of neurons in many areas of the nervous system have com- 
plex topology, such as a small world topology [57], highly 



p-1 



S. Scarpettal,2 F. Giaccol,2 A. de Candia3,4,2 



connected hubs and modularity [35]. Up to now, the only 
nervous system to have been comprehensively mapped at 
a cellular level is the one of Caenorhabditis elegans, and it 
has been found that is has indeed a small world structure 
27,29]. The same property was found for functional and 
anatomical connectivity, in different animals and areas of 

the brain HHGHIS]. 

In this paper, we focus on the functioning of the network 
as a dynamical associative memory, that is on the ability 
of the network to memorize and recall multiple dynamical 
patterns, where each pattern is characterized not by a set 
of binary states of the neurons, as in the Hopfield model, 
but rather by a different set of time shifts (phases) between 
the periodic activities of the neurons. We study how the 
capacity of the network depends on the number of neurons, 
on the number of connections, and on the topology of the 
network, that is on the distribution of the connections 
between neighboring or distant neurons. 

We consider a network composed by N neurons, with 
N(N— 1) possible (directed) connections Jy. The activity 
of the neuron is represented by a time-dependent variable 
<7i(t), with <Ji(t) = — 1 (silent neuron) or <7j(t) = 1 (firing 
neuron). Note that, in a coarse grained view, the variable 
<7i(t) may as well represent a group of neighboring neurons 
with a highly correlated activity. 

Not all the connections are present in the network. We 
put neurons randomly in a three-dimensional box with 
periodic boundary conditions, with density equal to one. 
For each neuron, we consider the sphere centered on it, 
with a radius R such that the sphere contains the z nearest 
neurons. Then we connect each neuron to (1 — 7)2; neurons 
chosen randomly within the sphere of the neighbors, and 
72: neurons chosen randomly in the whole system. In this 
way, we realize a network with a given mean connectivity 
z, and a given fraction 7 of long range vs. short range 
connections. 

During the learning phase, we force the network to re- 
produce a number P of spatio-temporal periodic patterns, 
given by a specified function 



(1) 



where fi = 1 ... P is the index of the pattern, uj^ 1 the an- 
gular velocity, c6f the phases of the neurons, that encode 
the relative times at which neurons start to fire in the pat- 
tern /x, and the function /(</>) is periodic of period 2tt. We 
consider the function 



f(4>) = 



1 if 
-1 if 



< , 

7T < 



mod 2tt < it, 
> mod 2tt < 2-7T, 



(2) 



with equal times of silent and firing state. The connection 
Jij represents the strength of the synapse going from neu- 
ron j to neuron i, or in the coarse grained view the sum 
of strengths of synapses going from the group of neurons 
j to the group of neurons i. While in the Hopfield model 
the pattern to be stored erf is static, and the learning 
rule is the outer product Jy cx erf a^, here the pattern to 




Fig. 1: The learning window A(t) used in the learning rule 
to model STDP, introduced and motivated by [32], A(t) 
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be stored erf (t) is time dependent, and we formulate the 
change in the connections in analogy with the STDP, 

as 

*max *max 

5Jij = J dt J d£ a % {t)A{t-t')<jj{t'), (3) 


where A(t) is the learning function, and [0, t max ] is the 
learning time interval in which the network is forced to 
reproduce the pattern fj,. We use for the function A(t) the 
one introduced and motivated by [35] , with the parameters 
that fit the experimental data of [22] (see Fig.[T]). In terms 
of Fourier components, this is written as 



(4) 



with /„ = ± and A(u) = jdtA(t)e^. 

Due to the temporal asymmetry of A(r), the Fourier 
component A(nw M ) has an imaginary part, and there- 
fore 5J^j 7^ SJj { . When we store multiple patterns /U = 
1,2, ... ,P, the learned weights are the sum of the con- 
tributions from individual patterns. For the sake of sim- 
plicity, we consider the same learning time i ma x and input 
frequency w* 1 for all the encoded patterns. Moreover, for 
each pattern \i, we extract the phases </>f randomly and 
uniformly from the interval [0, 2?r). 

After the learning phase, we initialize the network with 
a given initial condition <7i(0), and perform a zero temper- 
ature spontaneous dynamics, 



<Ti(t 



sign 



(5) 



with r the unit step of time. Due to the shape of the 
response function sign(x), if all the connections Jy are 
multiplied by the same positive constant, the dynamics 
is unchanged. Therefore, the learning time £ max and the 
amplitude of the learning function A(t) are immaterial. 
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Fig. 2: a) Plot of the self-sustained dynamics of the fully con- 
nected network with iV = 500 and P = 5. The dynamics of 
the network, after a transient, is periodic of period T = 8r. 
Different colors represent the value of \4ti/T\ mod 4, where ti 
is time at which the neuron i start to fire (black if neuron is 
firing at t = 0). In this case the times t, axe highly correlated 
with the phases cf>} , which means that the pattern fi = 1 is well 
retrieved, and the overlap mi (t) is large, b) The same network 
with P = 50. In this case the pattern is not retrieved, and the 
overlap is of order i/l/iV . 

In order to measure the similarity between the network 
activity during retrieval mode, and the stored phase-coded 
pattern /i, we define the overlap 

1 N 

m^t) = -^i{ty^. (6) 
1=1 

When the retrieval is perfect, <Ji(t) = f{ujt — </>f ), with an 
output angular velocity u that is in general different from 
the input u>^. In this case m^(t) = fie zut , where |/i| — 
0.64 for the function @. When the retrieval is not perfect, 
the modulus | m^{t)\ after a transient goes to a constant 
value lower than the maximal one. Finally, if the network 
is not able to reproduce the pattern, the overlap becomes 
after a transient of order 1/y/W. The modulus of m^(t) is 
therefore an order parameter which measure how much the 
network dynamics match the stored phase-coded pattern 
fi. Note that the output frequency w/27r depends on the 
input frequency lu^ /2tt, and on the degree of asymmetry 
of the learning function A(t). If uj is different from 
and |m^(i)| is high, it means that the phase-coded stored 
pattern /.t is replayed at a frequency different from the 
one used to store it, i.e. at a different time scale, but 
with the same phase relationship. In Fig. we show the 
dynamics of a network oi N = 500 neurons, with all the 
N(N — 1) connections activated, and with P = 5 patterns 
encoded at input frequency uj^ /2ix = 10 Hz. The network 
is initialized with a high overlap with the pattern fi = 1, 
setting 

V ' \ -1 if 7T < (f>\ <2tT. W 

Each segment represent a time interval in which the neu- 
ron i is firing, that is Ui(t) = 1, with the neurons ordered 



Fig. 3: a) Storage capacity of the network, as a function of 
the fraction of connections z/N and the number of patterns 
per neuron P/N, in the region of low connectivity, with N = 
40000 neurons. Solid (empty) circles represent points where 
the network is (is not) able to retrieve patterns. The network 
with N — 4000 gives the same result (within errors). In this 
region of low z/N the capacity is proportional to the number 
of connections, b) Storage capacity in the entire range of z/N, 
with TV = 4000 neurons. For clarity only a spline separating 
the encodable and the not-encodable region is shown. Inset: 
comparison between the N = 4000 and the N = 40000 cases for 
two values of z/N. The dashed line in all figures corresponds 
to P = 0.24z. 

on the vertical axis by the value of the phase of the first 
pattern <f>\. In this case the pattern fj, = 1 has been re- 
trieved, and the overlap m 1 ^) has modulus |m 1 (i)| ~ 0.63 
at long times. In Fig. [2b we show the same network with 
P = 50 patterns encoded. In this case the pattern is not 
retrieved, and the overlap has modulus |m 1 (t)| ~ 0.07. 

We define the capacity P ma x of the network as the max- 
imum number of patterns encodable in the network, and 
retrievable with an overlap greater than 0.2. We have first 
studied the capacity of the network as a function of the 
number of connections, in the case of a fraction 7 = 1 of 
long range connections. As the probability to create a long 
range connections is independent on the distance of the 
neurons, in this case the network is completely random. 
We therefore consider a network of TV neurons, and create 
a connection from each neuron to a number z of randomly 
chosen neurons, with < z < N. We then look if the net- 
work is able to encode P (random) pattern, and retrieve 
one of them subsequently. We initialize the network with 
an high overlap with one of the patterns, as in Eq. and 
simulate the dynamics in Eq. ([5]) up to a time t = 2 x 10 5 
in Monte Carlo steps, looking if the overlap with the cho- 
sen pattern remains higher than 0.2. The experiment is 
repeated with three different sets of P patterns. In Fig. [3^ 
the result is shown for N = 40000. Solid (empty) circles 
represent points where the network was (was not) able to 
retrieve a pattern at least two times out of three. In prac- 
tice, for all points considered, the network either was al- 
ways able to retrieve the pattern (three times out of three), 
or never (zero times out of three) . Furthermore, retrieved 
patterns had always an overlap greater than 0.36, while 
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Fig. 4: a) Storage capacity as a function of the fraction of long 
range connections 7, for a fixed number of connections per node 
(dashed line) or for a fixed total cost of the connections (solid 
line), b) Relative clustering coefficient and path length as a 
function of 7. 

not retrieved ones had an overlap lower than 0.08. Re- 
sults for TV = 4000 were practically the same, with a small 
number of points near the separation between encodablc 
and not encodable region showing some fluctuations (re- 
trieving the pattern one or two times out of three) . Note 
that, in the range of values of z/N considered in Fig. [3^, 
the maximum capacity P max is well described by a linear 
function of the connectivity, P ma x = 0.24 z. 

In Fig. [5Jd, we show the results for the entire range of 
z/N , from zero to one. When z/N is of the order of unity, 
the number of connections is of order N 2 , so the simula- 
tion is too expensive for TV = 40000. We therefore study 
sistematically only the network with N = 4000, but com- 
pare it with the case N = 40000 for two values of z/N, 
finding a good agreement. For clarity we do not show 
the points simulated, but only a spline separating the en- 
codable and the not-encodable region. The dashed line 
shows the limit of the capacity for low connectivity z/N, 
-Pmax = 0.24 z. It can be seen that, when the connectivity 
grows, there is a saturation effect, so that for z ~ N one 
finds P max /N ~ 0.032, and therefore P max ~ 0.032 z. 

To study the effects of short and long range connectiv- 
ity, we have then analyzed the capacity of the network 
in the case of a fraction 7 of long range connections, and 
(1 — 7) of short range connections. The result is shown 
in Fig. g}i (dashed line), for N = 40000 and z = 180 
(z/N = 0.0045). Note that in this case the sphere con- 
taining the short range neurons has a radius R ~ 3.5, 
while the side of the cubic box is L ~ 34. The storage ca- 
pacity, that is the maximum encodablc number of pattern 
such that the retrieval gives an overlap greater than 0.2, 
goes from P max ~ 0.11 z for 7 = (only local connections) 
to P max — 0.24 z for 7 = 1 (random network). This shows 
that the number of connections is not the only parameter 
that determines the capacity of the network. Long range 
(random) connections are more effective than short range 
ones in encoding patterns. 

The topology of the biological networks has been shaped 



by the evolution, as a compromise between the effective- 
ness in realizing complex tasks, and cost minimizations. 
While long range connections are more effective than short 
range ones, they are of course more costly. The trade-off 
between these two requirements will produce a network 
with a finite fraction of long range connections. We there- 
fore introduce a parameter /, that represents the cost of a 
long range connections with respect to a short range one, 
and we consider a network with (1 — 7)2; local connections, 
with neurons chosen randomly within those at distance 
lower than R, and 72// long range connections, with neu- 
rons chosen randomly in the whole system. Varying 7, in 
this case it is not the connectivity that is constant, but 
the total cost of the connections. The storage capacity 
as a function of 7 is shown in Fig. [4^ (solid line), for the 
case / = 3. The maximum capacity of the network is real- 
ized with 7 ~ 0.5, that corresponds to about 25% of long 
range connections over the total of the connections. To 
characterize the network, we have also calculated for the 
different values of 7 the clustering coefficient C, defined 
as the probability that two neurons, that are connected to 
a third neuron, are themselves connected, and the mean 
path length A, defined as the minimum number of connec- 
tions needed to go from a node to another node, averaged 
over all pairs of nodes. In Fig|4p, we show the relative 
clustering coefficient (C — C\ )/(Co — C\ ) , and path length 
(A — Ai)/(Aq — Ai), where the quantities with subscripts 
and 1 refer to the cases 7 = and 7 = 1 respectively. Note 
that the clustering coefficient decreases much more slowly 
than the path length, giving rise to a large region where 
the clustering coefficient is not much lower than that for 
local connections only, while the path length is almost as 
low as in the random network. The network therefore, in 
the region of intermediate 7 that corresponds to optimal 
capacity, is of a small- world type [2"7] . 

In this paper we studied the storage and recall of pat- 
terns in which information is encoded in the phase-based 
timing of firing relative to the cycle. We proposed a 
STDP-based learning rule, and we analyzed the its abil- 
ity to memorize multiple phase-coded patterns, such that 
the spontaneous dynamics of the network selectively gives 
sustained activity which matches one of the stored phase- 
coded patterns, depending on the initialization of the net- 
work. Our work generalizes the Hopfield model, to dynam- 
ical periodic states, characterized by the relative phases of 
the neurons. 

We have studied the storage capacity for different de- 
grees of sparseness and topologies of the connections. 
Changing the proportion 7 between short-range and long- 
range connections, we go from a three-dimensional net- 
work with only nearest-neighbors connections (7 = 0) to 
a random network (7 = 1). Small but finite values of 7 
give a "small world" topology, similar to that found in 
many areas of nervous system. 

We find that in the case of only short range connections 
the capacity is lowest, while in the case of only long range 
connections, that corresponds to a completely random net- 
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work, the capacity is highest. Moreover, a small but finite 
fraction of long range connection is enough to enhance 
the capacity highly, with respect to the short range case. 
Therefore if the cost of the connections is taken in account, 
with long range connections more costly than short range 
ones, than the optimal capacity will be given by a small 
fraction of long range connections, that corresponds to a 
small-world topology. 

This is in agreement with the observation that small- 
world attributes, with high clustering coefficient and short 
path length, was found across multiple spatial scales of 
cortical organization [3TJ[53]. This property, first found 
in C. elegans, is highly conserved over different type of 
measurement, and across different species, including cat, 
monkey and humans, for both functional and anatomical 
networks [25]. 
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